Scrolling and navigation in virtual reality

ABSTRACT

Various aspects of the subject technology relate to systems, methods, and machine-readable media for navigating through a shared artificial reality environment. Various aspects may include receiving an indication of a virtual object in the shared artificial reality environment. Aspects may also include receiving an input gesture indicative of a navigation command associated with the virtual object. Aspects may also include determining at least one type of the input gesture comprising flexion and extension, pronation and supination, or radial and ulnar. Aspects may also include determining a control method. Aspects may include determining a scrolling parameter. Aspects may include identifying the navigation command based on the type of the input gesture, the control method, and the scrolling parameter. Aspects may include applying the navigation command to the virtual object.

CROSS REFERENCE TO RELATED APPLICATIONS

This present application claims the benefit of priority under 35 U.S.C.120 as a continuation of U.S. Pat. Application Serial No. 17/670,228,filed Feb. 11, 2022, now allowed, the disclosure of which is herebyincorporated by reference in its entirety for all purposes.

TECHNICAL FIELD

The present disclosure generally relates to navigation and scrolling forcomputer generated shared artificial reality environments, and moreparticularly to user scrolling through scrollable lists of virtualobjects in such environments.

BACKGROUND

Interaction in a computer generated shared artificial realityenvironment involves interaction with various types of artificialreality/virtual content, elements, and/or applications in the sharedartificial reality environment. Users of the shared artificial realityenvironment may desire to select between options presented in the sharedartificial reality environment. For example, a virtual object in theenvironment can include a scrollable list. An ability to naturallyscroll through the scrollable list and browse through content withnatural hand gestures in the shared artificial reality environment mayenhance the user experience with respect to user movement forcontrolling scrolling and navigation in the environment.

BRIEF SUMMARY

The subject disclosure provides for systems and methods for navigationthrough an artificial reality environment such as a shared virtualreality environment such as scrolling through virtual areas or objects.For example, users of the shared virtual reality environment may usemovements to scroll through a scrollable list of a particular virtualobject or area. The movements can be wrist movements or other suitablemovements. That is, users can move their wrists via a midair scrollingtechnique that may be reflected by a virtual representation of a hand inthe environment, for example. Various wrist movements may correspond todifferent provided types of input gestures, such as flexion andextension, pronation and supination, radial and ulnar, etc. A method ofscrolling may be performed based on selected wrist movements whichdefine a navigation command according to one or more control methodsand/or scrolling parameters (e.g., a transfer function to simulatemomentum when scrolling via input gestures). The various mechanisms forscrolling and/or navigation in the environment of the present disclosuremay improve scrolling or navigation through virtual content, such as byproviding more natural and intuitive scrolling techniques. Accordingly,users advantageously may experience improved virtual interface(s) fornavigation in artificial reality environments.

According to one embodiment of the present disclosure, acomputer-implemented method for navigating through a shared artificialreality environment is provided. The method includes determiningreceiving an indication of a virtual object in the shared artificialreality environment. The method also includes receiving, via a virtualinterface, an input gesture indicative of a navigation commandassociated with the virtual object. The method also includes determiningat least one type of the input gesture. The type of the input gesturecomprises at least one of: flexion and extension, pronation andsupination, or radial and ulnar. The method also includes determining acontrol method. The method also includes determining a scrollingparameter. The method also includes identifying the navigation commandbased on the type of the input gesture, the control method, and thescrolling parameter. The method includes applying the navigation commandto the virtual object.

According to one embodiment of the present disclosure, a system isprovided including a processor and a memory comprising instructionsstored thereon, which when executed by the processor, causes theprocessor to perform a method for navigating through a shared artificialreality environment. The method includes determining receiving anindication of a virtual object in the shared artificial realityenvironment. The method also includes receiving, via a virtualinterface, an input gesture indicative of a navigation commandassociated with the virtual object. The method also includes determiningat least one type of the input gesture. The type of the input gesturecomprises at least one of: flexion and extension, pronation andsupination, or radial and ulnar. The method also includes determining acontrol method. The method also includes determining a scrollingparameter. The method also includes identifying the navigation commandbased on the type of the input gesture, the control method, and thescrolling parameter. The method includes applying the navigation commandto the virtual object.

According to one embodiment of the present disclosure, a non-transitorycomputer-readable storage medium is provided including instructions(e.g., stored sequences of instructions) that, when executed by aprocessor, cause the processor to perform a method for navigatingthrough a shared artificial reality environment. The method includesdetermining receiving an indication of a virtual object in the sharedartificial reality environment. The method also includes receiving, viaa virtual interface, an input gesture indicative of a navigation commandassociated with the virtual object. The method also includes determiningat least one type of the input gesture. The type of the input gesturecomprises at least one of: flexion and extension, pronation andsupination, or radial and ulnar. The method also includes determining acontrol method. The method also includes determining a scrollingparameter. The method also includes identifying the navigation commandbased on the type of the input gesture, the control method, and thescrolling parameter. The method includes applying the navigation commandto the virtual object.

According to one embodiment of the present disclosure, a non-transitorycomputer-readable storage medium is provided including instructions(e.g., stored sequences of instructions) that, when executed by aprocessor, cause the processor to perform a method for navigatingthrough a shared artificial reality environment. The method includesdetermining receiving an indication of a virtual object in the sharedartificial reality environment. The method also includes receiving, viaa virtual interface, an input gesture indicative of a navigation commandassociated with the virtual object. The method also includes determiningat least one type of the input gesture. The type of the input gesturecomprises at least one of: flexion and extension, pronation andsupination, or radial and ulnar. The method also includes determining acontrol method. The method also includes determining a scrollingparameter. The method also includes generating, based on the scrollingparameter, a momentum of scrolling through a scrollable list of thevirtual object in the shared artificial reality environment according tothe input gesture. The method also includes identifying the navigationcommand based on the type of the input gesture, the control method, andthe scrolling parameter. The method includes applying the navigationcommand to the virtual object. The method includes selecting an item ofthe scrollable list based on the navigation command and a double pinchinput gesture.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 is a block diagram of a device operating environment with whichaspects of the subject technology can be implemented.

FIGS. 2A-2B are diagrams illustrating virtual reality headsets,according to certain aspects of the present disclosure.

FIG. 2C illustrates controllers for interaction with an artificialreality environment, according to certain aspects of the presentdisclosure.

FIG. 3 is a block diagram illustrating an overview of an environment inwhich some implementations of the present technology can operate.

FIG. 4 illustrates an example artificial reality wearable, according tocertain aspects of the present disclosure.

FIG. 5 is a block diagram illustrating an example computer system (e.g.,representing both client and server) with which aspects of the subjecttechnology can be implemented.

FIGS. 6-8 illustrate example views of user navigation in an artificialreality environment, according to certain aspects of the presentdisclosure.

FIG. 9 is an example flow diagram for navigation through a sharedartificial reality environment, according to certain aspects of thepresent disclosure.

FIG. 10 is a block diagram illustrating an example computer system withwhich aspects of the subject technology can be implemented.

In one or more implementations, not all of the depicted components ineach figure may be required, and one or more implementations may includeadditional components not shown in a figure. Variations in thearrangement and type of the components may be made without departingfrom the scope of the subject disclosure. Additional components,different components, or fewer components may be utilized within thescope of the subject disclosure.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth to provide a full understanding of the present disclosure. It willbe apparent, however, to one ordinarily skilled in the art, that theembodiments of the present disclosure may be practiced without some ofthese specific details. In other instances, well-known structures andtechniques have not been shown in detail so as not to obscure thedisclosure.

The disclosed system addresses a problem in artificial reality tied tocomputer technology, namely, the technical problem of responsiveness touser inputs for navigation within a computer generated shared artificialreality environment. The disclosed system solves this technical problemby providing a solution also rooted in computer technology, namely, byproviding natural navigation (e.g., scrolling) techniques to users ofthe artificial reality environment such as based on a sensed wristmovement input gesture. The disclosed system also improves thefunctioning of the computer used to generate the artificial realityenvironment because it enables the computer to improve communicationbetween an artificial reality compatible user device and the computer.The present invention is integrated into a practical application ofcomputer based graphical user interface enabled navigation and scrollingfor virtual areas, objects, and/or elements. In particular, thedisclosed system provides more responsive, natural, and effectivescrolling based on an improved quantity and/or quality of controlmethods/mechanisms that more quickly and accurately translate inputwrist gestures into user desired navigation commands in the artificialreality environment, such as scrolling through virtual list elements inthe environment.

Aspects of the present disclosure are directed to creating andadministering artificial reality environments. For example, anartificial reality environment may be a shared artificial realityenvironment, a virtual reality (VR), an augmented reality environment, amixed reality environment, a hybrid reality environment, a non immersiveenvironment, a semi immersive environment, a fully immersiveenvironment, and/or the like. The artificial environments may alsoinclude artificial collaborative gaming, working, and/or otherenvironments which include modes for interaction between various peopleor users in the artificial environments. The artificial environments ofthe present disclosure may provide elements that enable users tonavigate (e.g., scroll) in the environments via function expansions inthe user’s wrist, such as via pinching, rotating, tilting, and/or thelike. For example, the degree that the user’s wrist is tilted cancorrespond to how quickly a scrollable list is scrolled through in theartificial environments (e.g., more tilt results in faster scrollingwhile less tile results in slower scrolling). As used herein,“real-world” objects are non-computer generated and artificial or VRobjects are computer generated. For example, a real-world space is aphysical space occupying a location outside a computer and a real-worldobject is a physical object having physical properties outside acomputer. For example, an artificial or VR object may be rendered andpart of a computer generated artificial environment.

Embodiments of the disclosed technology may include or be implemented inconjunction with an artificial reality system. Artificial reality,extended reality, or extra reality (collectively “XR”) is a form ofreality that has been adjusted in some manner before presentation to auser, which may include, e.g., virtual reality (VR), augmented reality(AR), mixed reality (MR), hybrid reality, or some combination and/orderivatives thereof. Artificial reality content may include completelygenerated content or generated content combined with captured content(e.g., real-world photographs). The artificial reality content mayinclude video, audio, haptic feedback, or some combination thereof, anyof which may be presented in a single channel or in multiple channels(such as stereo video that produces a three-dimensional effect to theviewer). Additionally, in some implementations, artificial reality maybe associated with applications, products, accessories, services, orsome combination thereof, that are, e.g., used to create content in anartificial reality and/or used in (e.g., perform activities in) anartificial reality. The artificial reality system that provides theartificial reality content may be implemented on various platforms,including a head-mounted display (HMD) connected to a host computersystem, a standalone HMD, a mobile device or computing system, a “cave”environment or other projection system, or any other hardware platformcapable of providing artificial reality content to one or more viewers.

“Virtual reality” or “VR,” as used herein, refers to an immersiveexperience where a user’s visual input is controlled by a computingsystem. “Augmented reality” or “AR” refers to systems where a user viewsimages of the real-world after they have passed through a computingsystem. For example, a tablet with a camera on the back can captureimages of the real-world and then display the images on the screen onthe opposite side of the tablet from the camera. The tablet can processand adjust or “augment” the images as they pass through the system, suchas by adding virtual objects. AR also refers to systems where lightentering a users’ eye is partially generated by a computing system andpartially composes light reflected off objects in the real world. Forexample, an AR headset could be shaped as a pair of glasses with apass-through display, which allows light from the real-world to passthrough a waveguide that simultaneously emits light from a projector inthe AR headset, allowing the AR headset to present virtual objectsintermixed with the real objects the user can see. The AR headset may bea block-light headset with video pass-through. “Artificial reality,”“extra reality,” or “XR,” as used herein, refers to any of VR, AR, MR,or any combination or hybrid thereof.

Several implementations are discussed below in more detail in referenceto the figures. FIG. 1 is a block diagram of a device operatingenvironment 100 with which aspects of the subject technology can beimplemented. The device operating environment can comprise hardwarecomponents of a computing system 100 that can create, administer, andprovide interaction modes for a shared artificial reality environment(e.g., collaborative artificial reality environment) such as fornavigation and/or scrolling via XR elements. The interaction modes caninclude various modes for various input gestures, control modes,scrolling parameters, etc. for each user of the computing system 100. Invarious implementations, the computing system 100 can include a singlecomputing device or multiple computing devices 102 that communicate overwired or wireless channels to distribute processing and share inputdata.

In some implementations, the computing system 100 can include astand-alone headset capable of providing a computer created or augmentedexperience for a user without the need for external processing orsensors. In other implementations, the computing system 100 can includemultiple computing devices 102 such as a headset and a core processingcomponent (such as a console, mobile device, or server system) wheresome processing operations are performed on the headset and others areoffloaded to the core processing component. Example headsets aredescribed below in relation to FIGS. 2A-2B. In some implementations,position and environment data can be gathered only by sensorsincorporated in the headset device, while in other implementations oneor more of the non-headset computing devices 102 can include sensorcomponents that can track environment or position data, such as forimplementing computer vision functionality. Additionally oralternatively, such sensors can be incorporated as wrist sensors, whichcan function as a wrist wearable for detecting or determining user inputgestures. For example, the sensors may include inertial measurementunits (IMUs), eye tracking sensors, electromyography (e.g., fortranslating neuromuscular signals to specific gestures), time of flightsensors, light/optical sensors, and/or the like to determine the inputsgestures, how user hands/wrists are moving, and/or environment andposition data.

The computing system 100 can include one or more processor(s) 110 (e.g.,central processing units (CPUs), graphical processing units (GPUs),holographic processing units (HPUs), etc.) The processors 110 can be asingle processing unit or multiple processing units in a device ordistributed across multiple devices (e.g., distributed across two ormore of computing device 102 s). The computing system 100 can includeone or more input devices 104 that provide input to the processors 110,notifying them of actions. The actions can be mediated by a hardwarecontroller that interprets the signals received from the input device104 and communicates the information to the processors 110 using acommunication protocol. As an example, the hardware controller cantranslate signals from the input devices 104 to simulate click moment orflip momentum with respect to XR scrolling, such as based on a transferfunction. Each input device 104 can include, for example, a mouse, akeyboard, a touchscreen, a touchpad, a wearable input device (e.g., ahaptics glove, a bracelet, a ring, an earring, a necklace, a watch,etc.), a camera (or other light-based input device, e.g., an infraredsensor), a microphone, and/or other user input devices.

The processors 110 can be coupled to other hardware devices, forexample, with the use of an internal or external bus, such as a PCI bus,SCSI bus, wireless connection, and/or the like. The processors 110 cancommunicate with a hardware controller for devices, such as for adisplay 106. The display 106 can be used to display text and graphics.In some implementations, the display 106 includes the input device aspart of the display, such as when the input device is a touchscreen oris equipped with an eye direction monitoring system. In someimplementations, the display is separate from the input device. Examplesof display devices are: an LCD display screen, an LED display screen, aprojected, holographic, or augmented reality display (such as a heads-updisplay device or a head-mounted device), and/or the like. Other I/Odevices 108 can also be coupled to the processor, such as a network chipor card, video chip or card, audio chip or card, USB, firewire or otherexternal device, camera, printer, speakers, CD-ROM drive, DVD drive,disk drive, etc.

The computing system 100 can include a communication device capable ofcommunicating wirelessly or wire-based with other local computingdevices 102 or a network node. The communication device can communicatewith another device or a server through a network using, for example,TCP/IP protocols. The computing system 100 can utilize the communicationdevice to distribute operations across multiple network devices. Forexample, the communication device can function as a communicationmodule. The communication device can be configured to transmit orreceive input gestures for determining navigation commands in XRenvironments or for XR objects (e.g., comprising scrollable lists).

The processors 110 can have access to a memory 112, which can becontained on one of the computing devices 102 of computing system 100 orcan be distributed across one of the multiple computing devices 102 ofcomputing system 100 or other external devices. A memory includes one ormore hardware devices for volatile or non-volatile storage, and caninclude both read-only and writable memory. For example, a memory caninclude one or more of random access memory (RAM), various caches, CPUregisters, read-only memory (ROM), and writable non-volatile memory,such as flash memory, hard drives, floppy disks, CDs, DVDs, magneticstorage devices, tape drives, and so forth. A memory is not apropagating signal divorced from underlying hardware; a memory is thusnon-transitory. The memory 112 can include program memory 114 thatstores programs and software, such as an operating system 118, XR worksystem 120, and other application programs 122 (e.g., XR games). Thememory 112 can also include data memory 116 that can include informationto be provided to the program memory 114 or any element of the computingsystem 100.

Some implementations can be operational with numerous other computingsystem environments or configurations. Examples of computing systems,environments, and/or configurations that may be suitable for use withthe technology include, but are not limited to, XR headsets, personalcomputers, server computers, handheld or laptop devices, cellulartelephones, wearable electronics, gaming consoles, tablet devices,multiprocessor systems, microprocessor-based systems, set-top boxes,programmable consumer electronics, network PCs, minicomputers, mainframecomputers, distributed computing environments that include any of theabove systems or devices, and/or the like.

FIGS. 2A-2B are diagrams illustrating virtual reality headsets,according to certain aspects of the present disclosure. FIG. 2A is adiagram of a virtual reality head-mounted display (HMD) 200. The HMD 200includes a front rigid body 205 and a band 210. The front rigid body 205includes one or more electronic display elements such as an electronicdisplay 245, an inertial motion unit (IMU) 215, one or more positionsensors 220, locators 225, and one or more compute units 230. Theposition sensors 220, the IMU 215, and compute units 230 may be internalto the HMD 200 and may not be visible to the user. In variousimplementations, the IMU 215, position sensors 220, and locators 225 cantrack movement and location of the HMD 200 in the real world and in avirtual environment in three degrees of freedom (3DoF), six degrees offreedom (6DoF), etc. For example, the locators 225 can emit infraredlight beams which create light points on real objects around the HMD200. As another example, the IMU 215 can include, e.g., one or moreaccelerometers, gyroscopes, magnetometers, other non-camera-basedposition, force, or orientation sensors, or combinations thereof. One ormore cameras (not shown) integrated with the HMD 200 can detect thelight points, such as for a computer vision algorithm or module. Thecompute units 230 in the HMD 200 can use the detected light points toextrapolate position and movement of the HMD 200 as well as to identifythe shape and position of the real objects surrounding the HMD 200.

The electronic display 245 can be integrated with the front rigid body205 and can provide image light to a user as dictated by the computeunits 230. In various embodiments, the electronic display 245 can be asingle electronic display or multiple electronic displays (e.g., adisplay for each user eye). Examples of the electronic display 245include: a liquid crystal display (LCD), an organic light-emitting diode(OLED) display, an active-matrix organic light-emitting diode display(AMOLED), a display including one or more quantum dot light-emittingdiode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER,etc.), some other display, or some combination thereof.

In some implementations, the HMD 200 can be coupled to a core processingcomponent such as a personal computer (PC) (not shown) and/or one ormore external sensors (not shown). The external sensors can monitor theHMD 200 (e.g., via light emitted from the HMD 200) which the PC can use,in combination with output from the IMU 215 and position sensors 220, todetermine the location and movement of the HMD 200.

FIG. 2B is a diagram of a mixed reality HMD system 250 which includes amixed reality HMD 252 and a core processing component 254. The mixedreality HMD 252 and the core processing component 254 can communicatevia a wireless connection (e.g., a 60 GHz link) as indicated by the link256. In other implementations, the mixed reality system 250 includes aheadset only, without an external compute device or includes other wiredor wireless connections between the mixed reality HMD 252 and the coreprocessing component 254. The mixed reality system 250 may also includea wrist wearable, such as for converting wrist input gestures intonavigation commands for scrolling in XR environments. The mixed realityHMD 252 includes a pass-through display 258 and a frame 260. The frame260 can house various electronic components (not shown) such as lightprojectors (e.g., LASERs, LEDs, etc.), cameras, eye-tracking sensors,MEMS components, networking components, etc. The electronic componentsmay be configured to implement computing vision-based hand tracking fortranslating hand movements and positions to XR navigation commands.

The projectors can be coupled to the pass-through display 258, e.g., viaoptical elements, to display media to a user. The optical elements caninclude one or more waveguide assemblies, reflectors, lenses, mirrors,collimators, gratings, etc., for directing light from the projectors toa user’s eye. Image data can be transmitted from the core processingcomponent 254 via link 256 to HMD 252. Controllers in the HMD 252 canconvert the image data into light pulses from the projectors, which canbe transmitted via the optical elements as output light to the users’eye. The output light can mix with light that passes through the display258, allowing the output light to present virtual objects that appear asif they exist in the real-world.

Similarly to the HMD 200, the HMD system 250 can also include motion andposition tracking units, cameras, light sources, etc., which allow theHMD system 250 to, e.g., track itself in 3DoF or 6DoF, track portions ofthe user (e.g., hands, feet, head, or other body parts), map virtualobjects to appear as stationary as the HMD 252 moves, and have virtualobjects react to gestures and other real-world objects For example, theHMD system 250 can track the motion and position of user’s wristmovements as input gestures for performing navigation such as scrollingof XR objects in a manner that is mapped to the input gestures. As anexample, the HMD system 250 may include a coordinate system to track therelative hand positions for each user for determining how the userdesires to scroll through the artificial reality environment with XRscrolling. In this way, the HMD system 250 can enable users to have anatural response and intuitive sense of controlled navigation andscrolling with their hands. The hand-based scrolling can be based on asingle control method or a hybrid of multiple control methods, such as acombination of flick and drag position control, for example.

FIG. 2C illustrates controllers 270 a-270 b, which, in someimplementations, a user can hold in one or both hands to interact withan artificial reality environment presented by the HMD 200 and/or HMD250. The controllers 270 a-270 b can be in communication with the HMDs,either directly or via an external device (e.g., core processingcomponent 254). The controllers can have their own IMU units, positionsensors, and/or can emit further light points. The HMD 200 or 250,external sensors, or sensors in the controllers can track thesecontroller light points to determine the controller positions and/ororientations (e.g., to track the controllers in 3DoF or 6DoF). Thecompute units 230 in the HMD 200 or the core processing component 254can use this tracking, in combination with IMU and position output, tomonitor hand positions and motions of the user. For example, the computeunits 230 can use the monitored hand positions to implement positioncontrol, rate control, nudges, and/or a combination thereof through ascrollable list. As an example, the compute units 230 can calculate aparameter for traversal through items of the scrollable list.

Position control may refer to the angle of a user’s hand being used toset a position of the scrollable list. The compute units 230 maycalculate start and release of list traversal, speed of scrollingthrough the list, and momentum of list movement. For example, momentumof list movement can refer to calculation of a transfer function by thecompute units 230 to simulate movement through the scrollable list witha simulated inertia of a user controlled “flick” through the list. Thecompute units 230 may, via the IMU outputs (or other sensor outputs viathe controllers 270 a-270 b), compute the change in position of theuser’s hand for defining an input gesture. For example, the computeunits 230 may implement computer vision/sensor-based hand tracking fordetermining that the user has made a pinch motion and moved down withtheir hand(s). Such a wrist motion can be defined as an input gesturethat is translated into a navigation command for selecting a scrollableXR object and scrolling downwards through a list of the XR object. Asdiscussed herein, the compute units 230 can support hybrid controlmethods that enables two or more different actions such as dragging andflicking for scrolling.

The compute units 230 can also be configured to implement a natural oran unnatural scrolling mode, which can be defined as simulated touchscreen scrolling or not. The compute units 230 may also apply a transferfunction in conjunction with sensed momentum from the controllers 270a-270 b. For example, the controllers 270 a-270 b can determine acorresponding momentum of a flicking gesture that the user makes withtheir hand, which can simulate the action of flicking an XR list in aparticular direction. The transfer function, such as a linear orquadratic transfer function, can define the momentum or inertia of howlong the flicked list moves in the particular direction before stopping.The compute units 230 can also compute the change in position of theuser’s hand for tracking other types of hand/wrist input gestures, suchas the user defining scrolling with a ring-based input gesture. That is,as the user moves their hand or finger in a circle, this may beconverted into a navigation command to scroll through the XR list. Thecompute units 230 can implement any combination of flexion andextension, pronation and supination, radial and ulnar input gestures.

The controllers 270 a-270 b can also include various buttons (e.g.,buttons 272A-F) and/or joysticks (e.g., joysticks 274A-B), which a usercan actuate to provide input and interact with objects. As discussedbelow, controllers 270 a-270 b can also have tips 276A and 276B, which,when in scribe controller mode, can be used as the tip of a writingimplement in the artificial reality environment. In variousimplementations, the HMD 200 or 250 can also include additionalsubsystems, such as a hand tracking unit, an eye tracking unit, an audiosystem, various network components, etc. to monitor indications of userinteractions and intentions. For example, in some implementations,instead of or in addition to controllers, one or more cameras includedin the HMD 200 or 250, or from external cameras, can monitor thepositions and poses of the users’ hands to determine gestures and otherhand and body motions. Such camera-based hand tracking can be referredto as computer vision, for example. Sensing subsystems of the HMD 200 or250 can be used to define motion (e.g., user hand/wrist motion) along anaxis (e.g., three different axes) along which the flexion and extension,pronation and supination, radial and ulnar input gestures can beperformed for determining corresponding navigation commands.

FIG. 3 is a block diagram illustrating an overview of an environment 300in which some implementations of the disclosed technology can operate.Environment 300 can include one or more client computing devices, suchas artificial reality device 302, mobile device 304, tablet 312,personal computer 314, laptop 316, desktop 318, and/or the like. Theartificial reality device 302 may be the HMD 200, HMD system 250, awrist wearable, or some other XR device that is compatible withrendering or interacting with an artificial reality or virtual realityenvironment. The artificial reality device 302 and mobile device 304 maycommunicate wirelessly via the network 310. In some implementations,some of the client computing devices can be the HMD 200 or the HMDsystem 250. The client computing devices can operate in a networkedenvironment using logical connections through network 310 to one or moreremote computers, such as a server computing device.

In some implementations, the environment 300 may include a server suchas an edge server which receives client requests and coordinatesfulfillment of those requests through other servers. The server mayinclude server computing devices 306 a-306 b, which may logically form asingle server. Alternatively, the server computing devices 306 a-306 bmay each be a distributed computing environment encompassing multiplecomputing devices located at the same or at geographically disparatephysical locations. The client computing devices and server computingdevices 306 a-306 b can each act as a server or client to otherserver/client device(s). The server computing devices 306 a-306 b canconnect to a database 308 or can comprise its own memory. Each servercomputing devices 306 a-306 b can correspond to a group of servers, andeach of these servers can share a database or can have their owndatabase. The database 308 may logically form a single unit or may bepart of a distributed computing environment encompassing multiplecomputing devices that are located within their corresponding server,located at the same, or located at geographically disparate physicallocations.

The memory of the server computing devices 306 a-306 b or the database308 can store scrolling or navigation information such as dataindicative of various control methods. For a particular XR device suchas a wrist sensor, a user may perform an input gesture such as aparticular wrist gesture that can be converted to a navigation (e.g.,scrolling) command according to a control method and/or scrollingparameter. The control method can be used to map a numerical angle(e.g., of the user’s hand relative to a scrollable list) to wristmovement determined by the wrist sensor. In particular, a positioncontrol method may refer to a control method in which the angle of thehand drives a position of the scrollable list. Rate control may refer toa joystick type control method in which the angle of the hand drives thevelocity of scrolling through the scrollable list. Another controlmethod may be based on discrete nudges from individual elements of thescrollable list such as moving one instance or items of the scrollinglist at a time per nudge, moving three items at a time, or moving someother discrete quantity based on a discrete nudge detected by the wristsensor via movement of the hand.

Various hybrids of control methods can be implemented by senseddetection of the user hand position and/or movement by the particular XRdevice. For example, the particular XR device may determine that thediscrete nudge control method should be triggered based on the relativeangle and/or movement of the hand and also determine that the ratecontrol method should be triggered if the hand makes a pinch and holdmotion. Either of the natural or hybrid control methods may also becombined with the scrolling parameter such as Boolean choices betweennatural or unnatural scrolling, stateless or stateful pinch, and/or thelike. Natural scrolling can refer to touchscreen type scrolling thattracks natural hand/finger motion while stateful pinch can refer toholding XR objects (e.g., the scrollable list) via the user hand controlwith a pinch and releasing the scrollable list with another pinch. Thatis, stateful pinch may enable XR users to experience the sensation andcontrol of holding and release XR elements via a single pinching motionwith their hands. The stateful or stateless parameter/characteristic ofthe input gesture can change how the input gesture is implemented andused to interpret corresponding navigation commands applied toartificial reality environments.

The network 310 can be a local area network (LAN), a wide area network(WAN), a mesh network, a hybrid network, or other wired or wirelessnetworks. The network 310 may be the Internet or some other public orprivate network. Client computing devices can be connected to network310 through a network interface, such as by wired or wirelesscommunication. The connections can be any kind of local, wide area,wired, or wireless network, including the network 310 or a separatepublic or private network. In some implementations, the server computingdevices 306 a-306 b can be used as part of a social network such asimplemented via the network 310. The social network can maintain asocial graph and perform various actions based on the social graph. Asocial graph can include a set of nodes (representing social networkingsystem objects, also known as social objects) interconnected by edges(representing interactions, activity, or relatedness). A socialnetworking system object can be a social networking system user,nonperson entity, content item, group, social networking system page,location, application, subject, concept representation or other socialnetworking system object, e.g., a movie, a band, a book, etc.

Content items can be any digital data such as text, images, audio,video, links, webpages, minutia (e.g., indicia provided from a clientdevice such as emotion indicators, status text snippets, locationindictors, etc.), or other multi-media. In various implementations,content items can be social network items or parts of social networkitems, such as posts, likes, mentions, news items, events, shares,comments, messages, other notifications, etc. Subjects and concepts, inthe context of a social graph, comprise nodes that represent any person,place, thing, or idea. A social networking system can enable a user toenter and display information related to the users’ interests, age /date of birth, location (e.g., longitude/latitude, country, region,city, etc.), education information, life stage, relationship status,name, a model of devices typically used, languages identified as onesthe user is familiar with, occupation, contact information, or otherdemographic or biographical information in the users’ profile. Any suchinformation can be represented, in various implementations, by a node oredge between nodes in the social graph.

A social networking system can enable a user to upload or createpictures, videos, documents, songs, or other content items, and canenable a user to create and schedule events. Content items can berepresented, in various implementations, by a node or edge between nodesin the social graph. A social networking system can enable a user toperform uploads or create content items, interact with content items orother users, express an interest or opinion, or perform other actions. Asocial networking system can provide various means to interact withnon-user objects within the social networking system. Actions can berepresented, in various implementations, by a node or edge between nodesin the social graph. For example, a user can form or join groups, orbecome a fan of a page or entity within the social networking system. Inaddition, a user can create, download, view, upload, link to, tag, edit,or play a social networking system object. A user can interact withsocial networking system objects outside of the context of the socialnetworking system. For example, an article on a news web site might havea “like” button that users can click. In each of these instances, theinteraction between the user and the object can be represented by anedge in the social graph connecting the node of the user to the node ofthe object. As another example, a user can use location detectionfunctionality (such as a GPS receiver on a mobile device) to “check in”to a particular location, and an edge can connect the user’s node withthe location’s node in the social graph.

A social networking system can provide a variety of communicationchannels to users. For example, a social networking system can enable auser to email, instant message, or text/SMS message, one or more otherusers. It can enable a user to post a message to the user’s wall orprofile or another user’s wall or profile. It can enable a user to posta message to a group or a fan page. It can enable a user to comment onan image, wall post or other content item created or uploaded by theuser or another user. And it can allow users to interact (via theiravatar or true-to-life representation) with objects or other avatars ina virtual environment (e.g., in an artificial reality workingenvironment), etc. In some embodiments, a user can post a status messageto the user’s profile indicating a current event, state of mind,thought, feeling, activity, or any other present-time relevantcommunication. A social networking system can enable users tocommunicate both within, and external to, the social networking system.For example, a first user can send a second user a message within thesocial networking system, an email through the social networking system,an email external to but originating from the social networking system,an instant message within the social networking system, an instantmessage external to but originating from the social networking system,provide voice or video messaging between users, or provide a virtualenvironment where users can communicate and interact via avatars orother digital representations of themselves. Further, a first user cancomment on the profile page of a second user or can comment on objectsassociated with a second user, e.g., content items uploaded by thesecond user.

Social networking systems enable users to associate themselves andestablish connections with other users of the social networking system.When two users (e.g., social graph nodes) explicitly establish a socialconnection in the social networking system, they become “friends” (or,“connections”) within the context of the social networking system. Forexample, a friend request from a “John Doe” to a “Jane Smith,” which isaccepted by “Jane Smith,” is a social connection. The social connectioncan be an edge in the social graph. Being friends or being within athreshold number of friend edges on the social graph can allow usersaccess to more information about each other than would otherwise beavailable to unconnected users. For example, being friends can allow auser to view another user’s profile, to see another user’s friends, orto view pictures of another user. Likewise, becoming friends within asocial networking system can allow a user greater access to communicatewith another user, e.g., by email (internal and external to the socialnetworking system), instant message, text message, phone, or any othercommunicative interface. Being friends can allow a user access to view,comment on, download, endorse or otherwise interact with another user’suploaded content items. Establishing connections, accessing userinformation, communicating, and interacting within the context of thesocial networking system can be represented by an edge between the nodesrepresenting two social networking system users.

In addition to explicitly establishing a connection in the socialnetworking system, users with common characteristics can be consideredconnected (such as a soft or implicit connection) for the purposes ofdetermining social context for use in determining the topic ofcommunications. In some embodiments, users who belong to a commonnetwork are considered connected. For example, users who attend a commonschool, work for a common company, or belong to a common socialnetworking system group can be considered connected. In someembodiments, users with common biographical characteristics areconsidered connected. For example, the geographic region users were bornin or live in, the age of users, the gender of users and therelationship status of users can be used to determine whether users areconnected. In some embodiments, users with common interests areconsidered connected. For example, users’ movie preferences, musicpreferences, political views, religious views, or any other interest canbe used to determine whether users are connected. In some embodiments,users who have taken a common action within the social networking systemare considered connected. For example, users who endorse or recommend acommon object, who comment on a common content item, or who RSVP to acommon event can be considered connected. A social networking system canutilize a social graph to determine users who are connected with or aresimilar to a particular user in order to determine or evaluate thesocial context between the users. The social networking system canutilize such social context and common attributes to facilitate contentdistribution systems and content caching systems to predictably selectcontent items for caching in cache appliances associated with specificsocial network accounts.

In particular embodiments, one or more objects (e.g., content or othertypes of objects) of a computing system may be associated with one ormore privacy settings. The one or more objects may be stored on orotherwise associated with any suitable computing system or application,such as, for example, a social-networking system, a client system, athird-party system, a social-networking application, a messagingapplication, a photo-sharing application, or any other suitablecomputing system or application. Although the examples discussed hereinare in the context of an online social network, these privacy settingsmay be applied to any other suitable computing system. Privacy settings(or “access settings”) for an object may be stored in any suitablemanner, such as, for example, in association with the object, in anindex on an authorization server, in another suitable manner, or anysuitable combination thereof. A privacy setting for an object mayspecify how the object (or particular information associated with theobject) can be accessed, stored, or otherwise used (e.g., viewed,shared, modified, copied, executed, surfaced, or identified) within theonline social network. When privacy settings for an object allow aparticular user or other entity to access that object, the object may bedescribed as being “visible” with respect to that user or other entity.As an example and not by way of limitation, a user of the online socialnetwork may specify privacy settings for a user-profile page thatidentifies a set of users that may access work-experience information onthe user-profile page, thus excluding other users from accessing thatinformation.

In particular embodiments, privacy settings for an object may specify a“blocked list” of users or other entities that should not be allowed toaccess certain information associated with the object. In particularembodiments, the blocked list may include third-party entities. Theblocked list may specify one or more users or entities for which anobject is not visible. As an example and not by way of limitation, auser may specify a set of users who may not access photo albumsassociated with the user, thus excluding those users from accessing thephoto albums (while also possibly allowing certain users not within thespecified set of users to access the photo albums). In particularembodiments, privacy settings may be associated with particularsocial-graph elements. Privacy settings of a social-graph element, suchas a node or an edge, may specify how the social-graph element,information associated with the social-graph element, or objectsassociated with the social-graph element can be accessed using theonline social network. As an example and not by way of limitation, aparticular concept node corresponding to a particular photo may have aprivacy setting specifying that the photo may be accessed only by userstagged in the photo and friends of the users tagged in the photo. Inparticular embodiments, privacy settings may allow users to opt in to oropt out of having their content, information, or actions stored/loggedby the social-networking system or shared with other systems (e.g., athird-party system). Although this disclosure describes using particularprivacy settings in a particular manner, this disclosure contemplatesusing any suitable privacy settings in any suitable manner.

In particular embodiments, privacy settings may be based on one or morenodes or edges of a social graph. A privacy setting may be specified forone or more edges or edge-types of the social graph, or with respect toone or more nodes, or node-types of the social graph. The privacysettings applied to a particular edge connecting two nodes may controlwhether the relationship between the two entities corresponding to thenodes is visible to other users of the online social network. Similarly,the privacy settings applied to a particular node may control whetherthe user or concept corresponding to the node is visible to other usersof the online social network. As an example and not by way oflimitation, a first user may share an object to the social-networkingsystem. The object may be associated with a concept node connected to auser node of the first user by an edge. The first user may specifyprivacy settings that apply to a particular edge connecting to theconcept node of the object, or may specify privacy settings that applyto all edges connecting to the concept node. As another example and notby way of limitation, the first user may share a set of objects of aparticular object-type (e.g., a set of images). The first user mayspecify privacy settings with respect to all objects associated with thefirst user of that particular object-type as having a particular privacysetting (e.g., specifying that all images posted by the first user arevisible only to friends of the first user and/or users tagged in theimages).

In particular embodiments, the social-networking system may present a“privacy wizard” (e.g., within a webpage, a module, one or more dialogboxes, or any other suitable interface) to the first user to assist thefirst user in specifying one or more privacy settings. The privacywizard may display instructions, suitable privacy-related information,current privacy settings, one or more input fields for accepting one ormore inputs from the first user specifying a change or confirmation ofprivacy settings, or any suitable combination thereof. In particularembodiments, the social-networking system may offer a “dashboard”functionality to the first user that may display, to the first user,current privacy settings of the first user. The dashboard functionalitymay be displayed to the first user at any appropriate time (e.g.,following an input from the first user summoning the dashboardfunctionality, following the occurrence of a particular event or triggeraction). The dashboard functionality may allow the first user to modifyone or more of the first user’s current privacy settings at any time, inany suitable manner (e.g., redirecting the first user to the privacywizard).

Privacy settings associated with an object may specify any suitablegranularity of permitted access or denial of access. As an example andnot by way of limitation, access or denial of access may be specifiedfor particular users (e.g., only me, my roommates, my boss), userswithin a particular degree-of-separation (e.g., friends,friends-of-friends), user groups (e.g., the gaming club, my family),user networks (e.g., employees of particular employers, students oralumni of particular university), all users (“public”), no users(“private”), users of third-party systems, particular applications(e.g., third-party applications, external websites), other suitableentities, or any suitable combination thereof. Although this disclosuredescribes particular granularities of permitted access or denial ofaccess, this disclosure contemplates any suitable granularities ofpermitted access or denial of access.

FIG. 4 illustrates an example artificial reality wearable, according tocertain aspects of the present disclosure. For example, the artificialreality wearables can be a wrist wearable such as a XR wrist sensor 400.The wrist sensor 400 may be configured to sense position and movement ofa user’s hand in order to translate such sensed position and movementinto input gestures. For example, the input gestures may be micromovements of the user’s wrist. As an example, the wrist movements mayinclude rotation, pinching, holding downward, holding upward, sliding,flicking, other suitable wrist movements, etc. The XR wrist sensor 400may generally represent a wearable device dimensioned to fit about abody part (e.g., a wrist) of the user. As shown in FIG. 4 , the XR wristsensor 400 may include a frame 402 and a sensor assembly 404 that iscoupled to frame 402 and configured to gather information about a localenvironment by observing the local environment. The sensor assembly 404can include cameras, IMU eye tracking sensors, electromyography (EMG)sensors, time of flight sensors, light/optical sensors, and/or the liketo track wrist movement.

In this way, the XR wrist sensor 400 can determine/detect the user inputgestures, interpret position and movement data of the user’s wrist, andconvert wrist input gestures to navigation commands based on specifiedcontrol method(s), scrolling parameter(s) and/or the like. The XR wristsensor 400 may also include one or more audio devices, such as outputaudio transducers 408 a-408 b and input audio transducers 410. Theoutput audio transducers 408 a-408 b may provide audio feedback and/orcontent to the user while the input audio transducers 410 may captureaudio in the user’s environment. The XR wrist sensor 400 may alsoinclude other types of screens or visual feedback devices (e.g., adisplay screen integrated into a side of frame 102). In someembodiments, the wrist wearable 400 can instead take another form, suchas head bands, hats, hair bands, belts, watches, ankle bands, rings,neckbands, necklaces, chest bands, eyewear frames, and/or any othersuitable type or form of apparatus. Other forms of the XR wrist sensor400 may be different wrist bands with a different ornamental appearancethan the XR wrist sensor 400 but perform a similar function for sensingwrist input gestures for XR navigation and scrolling in artificialreality environments.

FIG. 5 is a block diagram illustrating an example computer system 500(e.g., representing both client and server) with which aspects of thesubject technology can be implemented. The system 500 may be configuredfor navigating through a shared artificial reality environment,according to certain aspects of the disclosure. In some implementations,the system 500 may include one or more computing platforms 502. Thecomputing platform(s) 502 can correspond to a server component of anartificial reality/XR platform, which can be similar to or the same asthe server computing devices 306 a-306 b of FIG. 3 and include theprocessor 110 of FIG. 1 . The computing platform(s) 502 can beconfigured to store, receive, determine, and/or analyze user preferences(e.g., navigation preferences) and/or user information to improvescrolling and an overall user experience of the shared artificialreality environment. For example, the computing platform(s) 502 may beconfigured to execute algorithm(s) (e.g., mapping algorithms, transferfunction algorithms, machine learning algorithms etc.) to convert sensedwrist movements (e.g., flexion and extension, pronation and supination,radial and ulnar, etc.) such as via a computer vision-based user handtracking (e.g., HMD 200, HMD system 250) or via wrist movement trackingfrom a wrist wearable 400 a-400 b to implement navigation commands inthe shared artificial reality environment.

As an example, the wrist movements can be user micro gestures that aredetectable for implementing the navigation commands for scrollingrelative to XR elements such as an XR virtual area, XR object, virtualscrollable list, and/or the like. Such wrist movements and gestures canbe detected by a wrist gesture module 508. The computing platform(s) 502can maintain or store data, such as in the electronic storage 526, forrepresenting navigation (e.g., scrolling) techniques for enablingselections of various natural intuitive scrolling techniques for XRusers. Such scrolling techniques can include user hand gestures relativeto a fixed neutral point at a reference angle, such as pinching (e.g.,thumb-to-forefinger or other wrist pinch gesture above the fixed neuralpoint to nudge a scrollable list up once and thumb-to-forefinger pinchgesture below the point to nudge the scrollable list down once) tosimulate an arrow key function or to simulate grabbing the scrollablelist in place, increasing/decreasing scrolling speed via wrist gestureat increased/decreased angle (relative to reference angle) in acontinuous scrolling mode, simulate dragging the list by angling theuser hand and release the hand gesture with a particular velocity toflick the list, and/or the like. The scrolling techniques to simulatescrolling functions via wrist gestures described herein may beimplemented by mapping user hand angle (relative to referenceangle/fixed point) to navigation commands rather than using raycasts(e.g., raycast from the user’s hand to determine the position of areticle on an XR user interface to form a type of XR touch-screenfinger).

The computing platform(s) 502 may be configured to communicate with oneor more remote platforms 504 according to a client/server architecture,a peer-to-peer architecture, and/or other architectures. The remoteplatform(s) 504 may be configured to communicate with other remoteplatforms via computing platform(s) 502 and/or according to aclient/server architecture, a peer-to-peer architecture, and/or otherarchitectures. Users may access the system 500 hosting the sharedartificial reality environment and/or personal artificial reality viaremote platform(s) 504. In this way, the remote platform(s) 504 can beconfigured to cause output of a personalized version of the sharedartificial reality environment on client device(s) of the remoteplatform(s) 504, such as via the HMD 200, HMD system 250, and/orcontrollers 270 a-270 b of FIG. 2C. As an example, the remoteplatform(s) 504 can access artificial reality content and/or artificialreality applications for use in the shared artificial reality for thecorresponding user(s) of the remote platform(s) 504, such as via theexternal resources 524. The computing platform(s) 502, externalresources 524, and remote platform(s) 504 may be in communication and/ormutually accessible via the network 310.

The computing platform(s) 502 may be configured by machine-readableinstructions 506. The machine-readable instructions 506 may be executedby the computing platform(s) to implement one or more instructionmodules. The instruction modules may include computer program modules.The instruction modules being implemented may include one or more ofwrist gesture module 508, computer vision module 510, control methodmodule 512, pinch module 514, scrolling module 516, transfer functionmodule 518, XR module 520, and/or other instruction modules.

As discussed herein, the wrist gesture module 508 can convert inputgestures to navigation (e.g., scrolling commands) in an XR environment,such as based on wrist movement detection by a sensor component (e.g.,sensor assembly 404) of the remote platform(s) 504, such as for each XRcompatible device of the remote platforms(s) 504. The XR compatibledevice can be or include HMDs 200, 250, wrist wearables 400 a-400 b, orsome other type of XR applicable device. Based on sensed data from theremote platform(s) 504, the wrist gesture module 508 can determine anangle, relative coordinate, rotation, and/or other position/movementparameter of an input gesture, such as according to a coordinate systemimplemented by the computing platform(s) 502 (e.g., via XR module 520).For example, the sensed data can be IMU data, eye tracking data, EMGdata, time of flight data, optical data, and/or the like in order tocharacterize physical parameters of the input gesture such as the poseof the user’s hand when and during the making of the input gesture. Thewrist gesture module 508 can output a computerized rendering of theuser’s hand according to the actual sensed movement and position of thehand, which can be mapped to a navigation/scrolling command according tospecified control method(s) from the control method module 512 andscrolling parameter(s) from the scrolling module 516. As an example,wrist gestures may include flexion and extension, pronation andsupination, radial and ulnar, etc. Flexion can be visualized as thewrist being flexed at a downward 120° angle, for example. Pronation canbe visualized as rotation of the wrist, such as lateral rotation of thewrist when the hand is clenched into a fist. Ulnar deviation can bevisualized as a waving type motion of the wrist such as the wrist beingmoved to a rightward 45° angle with the fingers held together.

The computer vision module 510 may implement a similar function as thewrist wearable sensor performs, such as to track input gestures. Forexample, the computer vision module 510 can be part of a head mountedsensor (e.g., of HMD 200, 250) in order to optically track user handposition and movement, such as via an eye tracking component. Thecomputer vision module 510 can determine a relative position of variousparts of the user such as their hands, arms, legs, and/or the like. Inthis way, the computer vision module 510 can enable the XR module 520 togenerate an XR representation such as a user representation or avatar ofthe user in the shared artificial reality environment. In addition, thecomputer vision module 510 may generate sensed data such as based onvisual tracking on the user’s position and movement in the real world sothat changes in position and movement can be tracked and reflected inthe shared artificial reality environment by the computer vision module510. The computer vision module 510 can similarly track user positionand movement, such as of the user’s wrist, in order to implementnavigation commands such as scrolling of scrollable elements of XRelements based on how the wrist moves. In other words, the wristmovement can be an input gesture that is translated into a navigationcommand based on control method(s) and scrolling parameter(s).

The control method module 512 may specify one or more control methodsbeing used for translation of the navigation command. The controlmethods may include position control, rate control, discrete nudge,hybrid control methods, etc. A position control method may refer to acontrol method in which the angle of the hand drives a position of thescrollable list. Rate control may refer to a joystick type controlmethod in which the angle of the hand drives the velocity of scrollingthrough the scrollable list. As an example, the control methods may beused for scrolling for tasks such as rapidly scrolling and selectingitems in a familiar location, leisurely browsing continuous content,and/or the like. The posture of the user’s hand/wrist when making inputgestures according to specified control methods can be based on an armin front posture, arm at side posture, or some other suitable posture.The control methods may be specified by the control method module 512 toimplement various scrolling techniques by the scrolling module 516. Forexample, the scrolling techniques can include pinch displacement nudge,pinch velocity nudge, rate control, position control, stateful joysticktype A, stateful joystick type B, drag and flick, ring scrolling, and/orthe like.

The pinch nudge scrolling techniques can refer to using a displacementdistance of a user thumb-to-forefinger pinch motion or a speed of thepinch motion to control nudge a scrollable XR element, such as nudgingthe scrollable list one item down or up. The stateful joystick types canrefer to using particular defined relative angles of the wrist tocontrol scrolling and/or grabbing/releasing of the scrollable list. Dragand flick can refer to dragging an XR element with position control and“flicking” the XR element such as similar to the motion of a roulettewheel being spun. Ring scrolling may refer to the wrist being used tonavigate based on moving the hand in a circle to scroll up and down thescrollable list such as moving clockwise causes the list to scroll upwhile moving counterclockwise causes the list to scroll down. Thecontrol method module 512 can specify multiple control methods as acombination or hybrid of methods for the user to perform navigation andscrolling in XR areas of the artificial reality environment. Forexample, a hybrid control method may be position control until aspeed/velocity of the user’s wrist motion exceeds a certain threshold atwhich point an inertial flick is performed for implementation of adifferent control method. That is, the control method module 512 may beconfigured with a certain threshold for switching control methods suchas for performing the flick instead of the previously used controlmethod. For example, the configured thresholds can be speed thresholdsor displacement thresholds (e.g., navigation techniques at differentdisplacements). Additionally or alternatively, the control method module512 can be configured with multiple different types of gestures forhybrid control methods, such as short and long pinch, soft and firmpinch, etc.

The pinch module 514 may identify user pinching motions for conversionor determination of navigation commands according to selected orconfigured control method(s) and/or scrolling parameter(s). The userpinching motions can be detected by wrist wearables such as the wristsensors 400 a-400 b. For example, the wrist sensors 400 a-400 b maycomprise two IMUs, such as one IMU on the back of the user’s hand andone on the wrist. The IMUs can be configured to detect pinch such as viaa closed signal when the user pinches. For example, the IMUs can detectuser pinch movements down (e.g., in a downward direction) and user pinchmovements up (e.g., in an upward direction). Other pinch movements arealso possible to be detected in order for the user to control scrolling.As an example, the user may use the pinch as an input gesture, such asusing a pinch held in a specific direction to drag an XR element. Inparticular, the user can holding a pinch down to continuously scrolldown an XR scrollable object (e.g., scrollable list) or the user canmake discrete pinch motions to discretely move (e.g., discrete nudge)down instances of the list such as one pinch down causing the list tomove down by one item instance.

In this way, the user can use pinches as an input gesture for navigationcommands according to configured control method(s). For example, theuser can use hybrid control methods with pinching motions such as usingpinching to perform both dragging and flicking. The control method canbe switched between dragging and flicking according to a threshold, sopinches can be interpreted as both dragging and flicking commands. Theuser may also use a hybrid control method to scroll with both ratecontrol and position control. That is, both the angle and movement ofthe user’s hand can be used to set position and movement rate of thescrollable list. The pinch module 514 may also detect stateful orstateless pinches depending on whether such a statefulnesscharacteristic has been set as a scrolling parameter by the scrollingmodule 516. For stateful pinches, the pinch module 514 can interpret therespective use of pinch start and pinch release as performing differentfunctions. For example, the pinch module 514 can detect a pinch startgesture by the user to activate a scrolling technique and detect aseparate pinch release gesture by the user to stop scrolling thescrollable list, which can involve an XR simulation of holding thescrollable list by the user in the artificial reality environment. Forstateless pinches, only the pinch start gesture may be used. Forexample, the pinch module 514 may detect that the user makes the pinchstart gesture to toggle whether or not scrolling of the scrollable listis activated or pinch to nudge the scrollable list. The direction ofmovement or rotation of the user pinches can be according to thespecified wrist gestures such as flexing up and down for flexion,rotating about a circular motion (e.g., rotating a fist) for pronation,waving along a circle for ulnar deviation, circular hand movements, orany other suitable wrist gestures.

The scrolling module 516 may identify other configurable scrollingparameters or characteristics of XR scrolling. For example, thescrolling parameters can include stateful/stateless pinch,natural/unnatural scrolling, a type of transfer function (e.g., via thetransfer function module 518), selection mechanisms, and/or the like.The type of transfer function can be used for the scrolling module 516to determine a momentum of scrolling, such as a simulated movementinertia based on specific input gestures such as the user flicking thescrollable list. Selection mechanisms can refer to ways for the user toswitch between or select specific scrolling parameters. The scrollingmodule 516 may implement scrolling techniques that advantageouslysupport natural and intuitive one dimensional and two dimensional XRscrolling and selection in the shared artificial reality environment.The scrolling techniques described herein advantageously may work wellfor: discrete and continuous tasks, short and long scrolling distances,quick and slow scrolling tasks, minimal user interfaces, with the user’sarm at different positions (e.g., at the front or side), etc. Thescrolling module 516 can also make adjustments for errors in pinchdetection for the pinch module 514, such as based on uncertainty inpinch detection by IMUs (e.g., one IMU on the back of the hand and oneof the wrist) or other sensors of the wrist wearables 400 a-400 b (e.g.,wires connected to the band of the wrist wearables 400 a-400 b to closefor generating signals that are indicative of pinch motions). Somescrolling techniques may be more sensitive to errors in pinch detection,such as an adjustment to address potential errors in detection such asinaccurate detection of the user closing or finishing a particular pinchmotion. Stateful pinch detection can be more expressive based on theuser being able to grab and release via pinch motions. In general, thescrolling module 516 may map the angle or velocity of the user wristmotion to scrolling movement.

The transfer function module 518 may control output of the sharedartificial reality environment and personalized artificial realityenvironment. The transfer function module 518 can address noiseartifacts in the wrist wearables 400 a-400 b. After accounting fornoise, the transfer function module 518 can simulate a particularscrolling momentum, such as a click moment based on a wrist inputgesture made by the user. For example, if the user makes a flick inputgesture, the transfer function module 518 determines a moment ofscrolling such as through the scrollable list based on a “momentum”corresponding to the flick input gesture. The extent of the momentum ormomentum parameter can be determined by the transfer function module 518based on a transfer function. The transfer function module 518 canselect an optimal transfer function, such as a linear transfer function,quadratic transfer function, or other transfer that enables a more“controllable” momentum of scrolling through the scrollable list. As anexample, the optimal transfer function may consider balance, precision,and speed for users to scroll to specific points of the long scrollablelist. For example, the transfer function implemented by the transferfunction module 518 may enable users to make relatively large flickgestures at the beginning of scrolling through the list and then startslowing down as the items in the scrollable list reach the desiredpoint.

Prior to reaching the desired point, the transfer function module 518can apply the selected transfer function to cause the momentum ofscrolling that slows down and is more controllable by the user. Theselected transfer function could be based on predictive knowledge by thetransfer function module 518 of where the user is likely to stop in thepredictive list, which could be based on past user history (e.g.,previous user scrolling), characteristics of the scrollable list,scrolling context, machine learning algorithms, and/or the like. Ingeneral, the transfer function module 518 can improve the application oftransfer functions to scrolling by enabling slower flicks to lead toslower momentum and faster flicks to lead to faster momentum. Thetransfer function module 518 advantageously may improve the userexperience of scrolling by determining a more intuitive and naturalscrolling experience corresponding to a better selected transferfunction. In particular, the selected transfer function may balance howthe transfer function module 518 translates or converts the particularspeed of a flick made by the user to a particular level of momentum. Theselected transfer function may correspond to the context of thescrollable list, such as what items are contained in the list, whetherthe list is being scrolled in as part of an XR game, how the items ofthe list are semantically different, etc. The transfer function appliedby the transfer function module 518 can allow users to navigate orscroll in XR environments in a similar intuitive fashion to how theyscroll on touch screen devices.

The XR module 520 may be used to render the shared artificial realityenvironment for remote platform(s) 504 via the computing platform(s)502, for example. The XR module 520 may generate XR representations ofnavigation or scrolling actions, such as scrollbars, arrow keys, and/orthe like. Such XR representations can be temporarily rendered as XRvisual elements or not be rendered at all. The XR module 520 may alsocause the user to receive scrolling feedback, such as visual, haptic, orother types of signal to indicate when a navigation or scrolling commandis being performed. For example, the XR module 520 could cause the wristwearable 400 a-400 b or other XR compatible device to vibrate when theuser makes a pinch motion to start holding an XR element or anotherpinch motion to release the XR element. The XR module 520 may alsoprovide XR visual elements that track or indicate the types or othercharacteristics of input gestures made by users.

As an example, the XR visual elements can include a ring XR element thatreflects wrist “ring motion” as the user’s wrist is rotated about acircular motion to scroll or navigate. As an example, the XR visualelements can include a hand XR element that reflects an open pinch,closed pinch, or in between pinch movement as the user makes pinch inputgestures for selecting navigation commands. The XR module 520 may renderXR objects for various navigation or scrolling tasks, such as discreteand continuous scrolling tasks, different scrolling distances (e.g.,joystick type nudges, 2D or 3D navigation inputs, etc.), navigatingthrough videos, etc. As an example, the XR module 520 and the scrollingmodule 516 may implement navigation and scrolling in XR applications(e.g., VR games) based on various input gestures. For example, the usercan use various wrist input gestures at varying angles and movements(e.g., flicking motions, dragging motions, etc.) as well as pinchingmotions in order to select and perform game functions. For example, fora magic themed XR game, the user can pinch to select a potion or performa specific wrist input gesture to perform a spell or teleportation. Theuser may also use specific input gestures with rate control and/orposition control (or any other technique described herein) to controlhow, when, and how fast the user moves between different XR areas of thegame or other XR applications (e.g., workspace-based applications).

In some implementations, the computing platform(s) 502, the remoteplatform(s) 504, and/or the external resources 524 may be operativelylinked via one or more electronic communication links. For example, suchelectronic communication links may be established, at least in part, viathe network 310 such as the Internet and/or other networks. It will beappreciated that this is not intended to be limiting, and that the scopeof this disclosure includes implementations in which the computingplatform(s) 502, the remote platform(s) 504, and/or the externalresources 524 may be operatively linked via some other communicationmedia.

A given remote platform 504 may include client computing devices, suchas artificial reality device 302, mobile device 304 tablet 312, personalcomputer 314, laptop 316, and desktop 318, which may each include one ormore processors configured to execute computer program modules. Thecomputer program modules may be configured to enable an expert or userassociated with the given remote platform 504 to interface with thesystem 500 and/or external resources 524, and/or provide otherfunctionality attributed herein to remote platform(s) 504. By way ofnon-limiting example, a given remote platform 504 and/or a givencomputing platform 502 may include one or more of a server, a desktopcomputer, a laptop computer, a handheld computer, a tablet computingplatform, a NetBook, a Smartphone, a gaming console, and/or othercomputing platforms. The external resources 524 may include sources ofinformation outside of the system 500, external entities participatingwith the system 500, and/or other resources. For example, the externalresources 524 may include externally designed XR elements and/or XRapplications designed by third parties. In some implementations, some orall of the functionality attributed herein to the external resources 524may be provided by resources included in system 500.

The computing platform(s) 502 may include the electronic storage 526, aprocessor such as the processors 110, and/or other components. Thecomputing platform(s) 502 may include communication lines, or ports toenable the exchange of information with a network and/or other computingplatforms. Illustration of the computing platform(s) 502 in FIG. 5 isnot intended to be limiting. The computing platform(s) 502 may include aplurality of hardware, software, and/or firmware components operatingtogether to provide the functionality attributed herein to the computingplatform(s) 502. For example, the computing platform(s) 502 may beimplemented by a cloud of computing platforms operating together as thecomputing platform(s) 502.

The electronic storage 526 may comprise non-transitory storage mediathat electronically stores information. The electronic storage media ofthe electronic storage 526 may include one or both of system storagethat is provided integrally (i.e., substantially non-removable) withcomputing platform(s) 502 and/or removable storage that is removablyconnectable to computing platform(s) 502 via, for example, a port (e.g.,a USB port, a firewire port, etc.) or a drive (e.g., a disk drive,etc.). The electronic storage 526 may include one or more of opticallyreadable storage media (e.g., optical disks, etc.), magneticallyreadable storage media (e.g., magnetic tape, magnetic hard drive, floppydrive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM,etc.), solid-state storage media (e.g., flash drive, etc.), and/or otherelectronically readable storage media. The electronic storage 526 mayinclude one or more virtual storage resources (e.g., cloud storage, avirtual private network, and/or other virtual storage resources). Theelectronic storage 526 may store software algorithms, informationdetermined by the processor(s) 110, information received from computingplatform(s) 502, information received from the remote platform(s) 504,and/or other information that enables the computing platform(s) 502 tofunction as described herein.

The processor(s) 110 may be configured to provide information processingcapabilities in the computing platform(s) 502. As such, the processor(s)110 may include one or more of a digital processor, an analog processor,a digital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information. Although theprocessor(s) 110 is shown in FIG. 1 as a single entity, this is forillustrative purposes only. In some implementations, the processor(s)110 may include a plurality of processing units. These processing unitsmay be physically located within the same device, or the processor(s)110 may represent processing functionality of a plurality of devicesoperating in coordination. Processor(s) 110 may be configured to executemodules 508, 510, 512, 514, 516, 518, 520, and/or other modules.Processor(s) 110 may be configured to execute modules 508, 510, 512,514, 516, 518, 520, and/or other modules by software; hardware;firmware; some combination of software, hardware, and/or firmware;and/or other mechanisms for configuring processing capabilities on theprocessor(s) 110. As used herein, the term “module” may refer to anycomponent or set of components that perform the functionality attributedto the module. This may include one or more physical processors duringexecution of processor readable instructions, the processor readableinstructions, circuitry, hardware, storage media, or any othercomponents.

It should be appreciated that although the modules 508, 510, 512, 514,516, 518, and/or 520 are illustrated in FIG. 5 as being implementedwithin a single processing unit, in implementations in which theprocessor(s) 110 includes multiple processing units, one or more of themodules 508, 510, 512, 514, 516, 518, and/or 520 may be implementedremotely from the other modules. The description of the functionalityprovided by the different modules 508, 510, 512, 514, 516, 518, and/or520 described herein is for illustrative purposes, and is not intendedto be limiting, as any of the modules 508, 510, 512, 514, 516, 518,and/or 520 may provide more or less functionality than is described. Forexample, one or more of the modules 508, 510, 512, 514, 516, 518, and/or520 may be eliminated, and some or all of its functionality may beprovided by other ones of the modules 508, 510, 512, 514, 516, 518,and/or 520. As another example, the processor(s) 110 may be configuredto execute one or more additional modules that may perform some or allof the functionality attributed below to one of the modules 508, 510,512, 514, 516, 518, and/or 520.

The techniques described herein may be implemented as method(s) that areperformed by physical computing device(s); as one or more non-transitorycomputer-readable storage media storing instructions which, whenexecuted by computing device(s), cause performance of the method(s); or,as physical computing device(s) that are specially configured with acombination of hardware and software that causes performance of themethod(s).

FIGS. 6-8 illustrate example views 600, 700, 800 of user navigation inan artificial reality environment, according to certain aspects of thepresent disclosure. The example view 600 may illustrate using an inputgesture for rate control as a control method based on a relative angleof a user’s wrist in the input gesture. The example view 700 mayillustrate using an input gesture for a hybrid control method includingnudge and rate control based on an absolute angle of the user’s wrist inthe input gesture. The example view 700 may illustrate using an inputgesture for a position control (drag) with flick as a hybrid controlmethod based on the position of the user’s wrist in the input gesture.The example views 600, 700, 800 show XR scrollable lists 602, 702, 802at various configurations; that is, the scrollable lists 602, 702, 802may be scrolled at different levels of a scrolling range. Accordingly,the scrollable lists 602, 702, 802 each show different instances or XRitems that are part of the scrollable lists 602, 702, 802. To navigatethe scrollable lists 602, 702, 802, the user may use an input gesture toscroll up or scroll down through the scrollable lists 602, 702, 802.

Although FIGS. 6-8 illustrate scrolling up and down, it should be notedthat other directions and types of scrolling or navigation are possible,such as scrolling to the left or right, scrolling in a circle, etc. Theinput gesture selected and/or performed by the user can be reflected orshown via the hand XR object 606, 706, 806. Additionally oralternatively, a hand can be rendered in non-XR formats. That is, handgestures can be detected without user interface intervention. The handgestures performed by the user can include any kind of system detectablespecific gesture or microgesture. A particular instance or XR item canbe selected via the hand XR object 606, 706, 806 and can be indicated byhighlighting the selected instance(s) of the scrollable lists 602, 702,802 such as the word “establish” in FIG. 7 . In some embodiments, thehighlighted word may be selected as a favorite word, which can bereflected in the XR text window 604, 704, 804. Alternatively, thehighlight can be used merely to indicate what would be selected if theuser performed a separate selection gesture. For example, in FIG. 6 ,the word “appoint” is highlighted in the scrollable list 602, but thecurrent favorite word is shown as “intend” in the XR text window 604. Ifthe user would make an input gesture that corresponds toa selectioncommand, the favorite word would change to “appoint.” The user may usean input gesture that corresponds to navigation/scrolling commands toscroll about the scrollable lists 602, 702, 802. The input gesture madeby the user may be converted to navigation/scrolling commands based onthe associated configured control method(s) and scrolling parameter(s).Such a conversion can be illustrated by the flexion/extension type inputgestures explained by Appendix A.

For example, the example view 600 of FIG. 6 may depict relative ratecontrol (e.g., a hybrid control method) being used. That is, the angleof a pinching motion made by the user as an input gesture may correspondto and/or be used to select a velocity of scrolling based on therelative difference between the angle (e.g., instantaneous angle) and aneutral point/threshold angle. The neutral point/threshold can bedetermined by the location of an earlier made pinch gesture. As anexample, the velocity of scrolling may be calculated based on thedifference between the hand’s current location and the hand’s locationwhen the user stared making a pinching motion. The example view 700 ofFIG. 7 may illustrate a displacement nudge with rate control. That is,each discrete pinching motion or held closed pinch by the user’s wristmay be used as an input gesture to scroll through the list via adiscrete nudge or for continuous scrolling, respectively. Eachindividual pinch can cause a discrete nudge of scrolling through thescrollable list 700 by one item, two items, or some other set discretenumber of items/instances while the held pinching motion can cause thescrollable list 700 to be continuously scrolled through until the userreleases the held pinching motion. For example, the example view 800 ofFIG. 8 may illustrate a drag and flick control being used. That is, adragging pinching motion can be used to scrolling through the list whilea flicking pinch motion can be used to flick the scrollable list through(e.g., as if the list were a roulette wheel being spun) its constituentitems with a moment controlled via an applicable transfer function, forexample. The dragging and flicking control may be intuitive but requiremore movement for scrolling compared to the navigation/scrollingcommands of FIG. 6 and FIG. 7 .

In addition, the pinching motion can be stateful or stateless, such asdepending on whether pinch start and release can be detected. Statefulpinching may refer to starting a pinch to hold the scrollable lists 602,702, 802 and release the pinch to release the scrollable lists 602, 702,802, while stateless pinching means using multiple pinches to hold orrelease the scrollable lists 602, 702, 802. Also, the pinching motioncan be used as discrete pinches to act as arrow keys such as in FIG. 7 .Furthermore, the user may either hold the pinch to scroll continuouslyor the user may pinch and displace further past or under a thresholddistance to activate or deactivate continuous scrolling. Alternativelyto the pinching motion, a similar arrow key-like technique could beimplemented using thresholds and no pinches. As an example, the user canmake a hand nudging motion up or down to cause a scrolling movement suchas nudging the list whenever the hand crosses a displacement threshold.If the user holds their hand in this displaced position for a definedquantity of time, this may cause continuous scrolling such as with ratecontrol.

Other types of user navigation are also possible and described herein.Specific steps of hand/wrist movements for user navigation are furtherillustrated and described in Appendix A. Types of user navigation andscrolling described herein can be a combination of interaction method(e.g., specific wrist gesture and control method combination/hybrid) andscrolling parameter considerations. As discussed herein, the interactionmethod can be a type of hand movement such as flexion/extension vspronation/supination vs radial/ulnar deviation. The scrolling parameterscan include settings such as snap on/off, natural/unnatural scrolling,transfer function type, and stateless/stateful pinch. Natural scrollingcan refer to whether scrollable content moves in the same direction asthe user’s hand while unnatural scrolling refers to scrollable contentmoving in the opposite direction as the user’s hand. Snap being on canrefer to the central item in the list being automatically highlightedand the scrollable lists 602, 702, 802 will also snap the highlighteditem into place when the scrollable lists 602, 702, 802 stops moving(snap being off turns off this highlight and snapping functionality).Stateful pinch can refer to controlling the scrollable lists 602, 702,802 while pinching is currently occurring and releasing when pinching isnot currently occurring. In contrast, stateless pinch can refer to thescrollable lists 602, 702, 802 switching from being controlled/held andreleased every time the user makes a pinching motion with their hand.The transfer function (e.g., linear or quadratic) can refer tosimulation of scrolling momentum, such as by mapping hand angle tovelocity. Many such settings can be adjusted to change the type ofnavigation applicable for the user. Some control methods introduceadditional settings and parameters. For example, rate control could usea relative or a fixed center and a linear or a quadratic transferfunction.

These settings can be used in any combination. For example, the user maymake a flexion/extension type of input gesture according to rate controlwith a quadratic transfer function. This scrolling method may have arelative center setting turned off with natural scrolling and snapsettings turned on. This can enable angle-based rate control. With thistechnique, the scrolling speed through the scrollable lists 602, 702,802 is increased the more the user’s hand is flexed/extended. Such ratecontrol mimics the function of a gas pedal by making scrolling speedgreater based on how flexed/extended the hand is. If unnatural scrollingis used and relative center is turned on with this scrolling method,then the rate control becomes relative rate control such that thescrolling speed through the scrollable lists 602, 702, 802 is greaterthe more that the hand is flexed/extended relative to where the userstarted a pinching motion (or displacement motion).

The quadratic transfer function may specify quadratic interpolation formapping hand angle to velocity while linear transfer functions specifylinear interpolation for the same motion. Another scrolling methodinvolves using pronation/supination type of input gesture according torate control with a linear transfer function. This scrolling method mayhave a relative center and snap settings turned off with unnaturalscrolling turned on. This may enable relative angle-based rate controlin which the velocity of scrolling is based on a relative angle of handflexing/extension relative to a baseline angle. The direction ofscrolling can be controlled based on the direction of userpronation/supination. As such, this scrolling method may be analogous toscrolling control via a gas knob. Another scrolling method may involvethe user making a radial/ulnar type of input gesture according to ratecontrol with a linear transfer function.

This scrolling method may have a relative center and snap settingsturned off with natural scrolling turned on. This can result in asimilar scrolling technique as the gas knob, but the scrolling velocitythrough the scrollable lists 602, 702, 802 is based on how much the handis deviated, such as similar to a gas lever in which the greater theflexing, the faster the scrolling speed. Another scrolling method mayinvolve the user making a flexion/extension type of input gestureaccording to position control with stateful pinch, natural scrolling,and without snap. That is, the user can hold and release the scrollablelists 602, 702, 802 without flicking being enabled. For example, theuser pinches to “grab” the list and releases the pinch to “drop thelist” while the scrollable lists 602, 702, 802 are displaced an amountproportion to how much the hand’s angle is displaced from where itstarted as part of angle-based position control. Another scrollingmethod may involve the user making a pronation/supination type of inputgesture according to position control with stateless pinch, naturalscrolling, and with snap. For example, the user may pinch to “grab” thelist and pinch again to “drop the list” while the scrollable lists 602,702, 802 are displaced amount proportional to how much the hand’s angleis displaced from where it started as part of angle-based positioncontrol. Such position control may require grabbing and releasing thelist. For example, if the scrollable lists 602, 702, 802 are relativelylong, the user may not be able to fully scroll through the list withintheir range of flexion/extension. In such a case, the user may berequired to grab and release the list multiple times as part ofrepeating the scrolling gesture.

Another scrolling method may involve the user making a radial/ulnar typeof input gesture according to position control with stateful pinch,unnatural scrolling, and with snap. Another scrolling method can involvethe user making a flexion/extension type of input gesture according toinertia control with natural scrolling and with snap. As an example,this angle-based inertia control can simulate the scrollable lists 602,702, 802 resting on a surface. Changing the hand’s angle “tilts” thesurface, causing the content of the scrollable lists 602, 702, 802 toslide in the corresponding direction. If the user’s hand returns toneutral, the list will continue moving for a while until simulated“friction” slows it to a stop; the simulated friction can bepredetermined such as based on a transfer function. To stop the movementof the scrollable lists 602, 702, 802 more quickly, the user can brieflyangle their hand in the opposite direction. Another scrolling method caninvolve the user making a pronation/supination type of input gestureaccording to inertia control with unnatural scrolling, and with snap on.Scrolling may again simulate a tilted surface such that changing thehand’s angle “tilts” the surface, causing the content of the scrollablelists 602, 702, 802 to slide in the corresponding direction. Anotherscrolling method may involve the user making a ring type of inputgesture according to position control with natural scrolling, and withsnap on.

That is, the user can move their finger in a rotational shape in arotational shape, such as drawing a circle in the air. With ring-basedposition control, the list is displaced an amount proportional to howdisplaced the hand is clockwise or counter-clockwise from where itstarted. Since the user’s hand can go around arbitrarily many times,grabbing or releasing the scrollable lists 602, 702, 802 is notnecessary. Based on natural or unnatural scrolling, the direction ofscrolling through the scrollable lists 602, 702, 802 can correspond toor be intuitively opposite to the ring-based hand rotation movement.Additionally or alternatively, the displacement of the scrollable lists602, 702, 802 can be a function of the hand’s rotational displacement.Also, unnatural scrolling can be turned on and snap can be turned off.Another scrolling method may involve the user making a flexion/extensiontype of input gesture according to a velocity nudge control method withstateless pinch. The velocity nudge control method may mean that thescrollable lists 602, 702, 802 are nudged by a discrete number ofinstances/items per nudge. For example, a pinch velocity nudge can causethe scrollable lists 602, 702, 802 to nudge up or down one time,although the nudges can be incremented by a different number such as twoitems per nudge, five items per nudge, and/or the like.

The user may flex or extend their hand in the pinch position to indicatethe direction of the nudge. The user may also flex or extend their handfaster than a certain threshold without pinching. This action may besimilar to waving the scrollable lists 602, 702, 802 along. The user mayalso use pronation/supination for the pinch-less velocity nudge such asbased on nudging the scrollable lists 602, 702, 802 whenever the userrotates their hand faster than a certain amount in the correspondingdirection. Another scrolling technique can involve the user making aflexion/extension type of pinch-less input gesture according to adisplacement nudge control method with natural scrolling and with snap.This pinch-less displacement nudge may cause the scrollable lists 602,702, 802 to nudge up or down by the specified nudge increment wheneverthe flex/extension angle of the user’s hand exceeds a certain positiveor negative threshold. Also, the user may use pronation/supination inputgestures for a pinched displacement nudge with unnatural scrolling andsnap. That is, each pinching motion can cause the scrollable lists 602,702, 802 to be nudged up or down based on which way the wrist isrotated. Scrolling techniques and methods can be hybrids involvinghybrids of the control methods discussed herein (e.g.,flexion/extension, pronation/supination, deviation, etc.).

One hybrid scrolling method may involve the user using aflexion/extension type of input gesture according to hybrid controlcomprising rate control and no-pinch displacement nudge for more preciseshort-distance scrolling, which can result in a type of joystick typescrolling. As an example, this joystick scrolling may involve a “neutralzone” of angles close to 0, “nudge zones” on either side of that, andall angle values beyond the “nudge zone” as “scroll zones.” If the userflexes/extends their hand into the nudge zone, this would cause thescrollable lists 602, 702, 802 to be nudged up or down by the specifiedincrement and continue nudging by the increment per second or otherspecified delay as long as the user’s hand is held in the nudge zone. Ifthe hand is flexed/extended even further, the scrollable lists 602, 702,802 may begin scrolling continuously with rate control. A similarjoystick scrolling technique uses pronation/supination as the inputtechnique. For example, this joystick scrolling may involve the userrotating their hand into the nudge zone and/or rotating even furtherpast a threshold rotation to cause continuous scrolling with ratecontrol.

Other joystick scrolling techniques may be implemented with statelesspinching. For example, the user may use flexion/extension as an inputgesture with natural scrolling and snap with the neutral zone and thescroll zone described herein. Due to the stateless pinchingconfiguration, the user would pinch to activate scrolling and pinchagain to deactivate scrolling in the scroll zone. For example, the usercan pinch to toggler scrolling from any zone. While scrolling, if theuser flexes/extends their hand into the scroll zone, the scrollablelists 602, 702, 802 will nudge according to the specified nudgeincrement. If the user holds their hand in the scroll zone, thescrollable lists 602, 702, 802 may begin scrolling with rate control.Such a flexion/extension stateless joystick type scrolling technique canalso be implemented with unnatural scrolling. As an example, for anotherjoystick scrolling technique, every new user pinch motion may define anew center point. That is, the user briefly nudging their hand up ordown after a pinch, and then returning it to center, will nudge the listonce by the specified increment. Additionally, pinching, moving thehand, and holding the hand there can result in the scrollable lists 602,702, 802 scrolling continuously in that direction until the hand returnsto the point of the pinch (e.g., the new center point).

As an example, another joystick scrolling technique may enable the userto pinch in the nudge zone to move the list one unit in the appropriatedirection. The user can pinch in the scroll zone to start scrolling thescrollable lists 602, 702, 802 with rate control until the user returnstheir hand to the center point. As an example, another joystickscrolling technique can enable the user to use a displacement nudge bypinching with the user’s hand angled up or down to nudge the scrollablelists 602, 702, 802 up or down. Multiple pinches can be used to nudgethe scrollable lists 602, 702, 802 multiple more times. After pinching,the user may angle the hand even further to start continuous scrollingof the scrollable lists 602, 702, 802 with rate control until the handreturns to where pinching started.

An alternative hybrid scrolling technique also uses flexion/extension asthe input gesture with a neutral zone and scroll zones similar todescribed above and the use of stateful pinch and unnatural scrolling.Accordingly, the user may pinch in the scroll zone to cause thescrollable lists 602, 702, 802 to nudge by the specified increment. Theuser can hold a pinching motion to begin scrolling the scrollable lists602, 702, 802 with rate control. Another hybrid scrolling techniqueapplies a “drag and flick” hybrid control method, such as for efficientlong-distance scrolling. The drag and flick control method may involvethe user performing a flexion/extension type of input gesture accordingto position control with stateful pinch, natural scrolling, snap, andadded flick functionality. As an example, with stateful pinchdrag/flick, the user can pinch and hold to “grab” the scrollable lists602, 702, 802 and drag it by changing the angle of their hand forposition control.

If the hand is moving faster than a certain rate when the user releases,the scrollable lists 602, 702, 802 will not stop but instead keep movingat its previous speed, gradually slowing with simulated “friction” suchas similar to or according to a specified transfer function. While thescrollable lists 602, 702, 802 are slowing down according to specifiedinertia (e.g., transfer function), the scrollable lists 602, 702, 802can be grabbed again by the user such as with another pinch, which canimitate touchscreen type scrolling. Grabbing the scrollable lists 602,702, 802 again while the scrollable lists 602, 702, 802 is moving withsimulated friction can be based on a held pinch or another pinch,depending on whether stateful or stateless pinching settings areapplied. In addition, the drag and flick navigation command can beperformed based on a pronation/supination type of input gesture.

FIG. 9 illustrates an example flow diagram (e.g., process 900) fornavigating through a shared artificial reality environment in a sharedartificial reality environment, according to certain aspects of thedisclosure. For explanatory purposes, the example process 900 isdescribed herein with reference to one or more of the figures above.Further for explanatory purposes, the steps of the example process 900are described herein as occurring in serial, or linearly. However,multiple instances of the example process 900 may occur in parallel. Forpurposes of explanation of the subject technology, the process 900 willbe discussed in reference to one or more of the figures above.

At step 902, an indication of a virtual object in the shared artificialreality environment may be received. According to an aspect, receivingthe input gesture comprises sensing a hand motion by a wrist mountedsensor. For example, the virtual object comprises at least one of: ascrollable list, a scrollable object, a virtual area, or a highlightedvirtual object. At step 904, an input gesture indicative of a navigationcommand associated with the virtual object may be received. For example,the first coordinate corresponds to a location of the userrepresentation in the shared artificial reality environment. Accordingto an aspect, sensing the hand motion comprises sensing a first motionby a first hand and a second motion by a second hand. The second motioncomprises a modification of the input gesture triggered by the firstmotion. According to an aspect, sensing the hand motion comprisessensing a first motion by a first hand via the virtual interface for thenavigation command and a second motion by a second hand via anothervirtual interface for another navigation command.

At step 906, at least one type of the input gesture may be determined.For example, the type of the input gesture comprises at least one of:flexion and extension, pronation and supination, or radial and ulnar.According to an aspect, determining the at least one type of the inputgesture comprises comparing a sensed motion by a wrist mounted sensorwith an optical signal from a head mounted sensor. At step 908, acontrol method may be determined. According to an aspect, determiningthe control method comprises determining at least one of: positioncontrol, rate control, or discrete nudge. At step 910, a scrollingparameter may be determined. According to an aspect, determining thescrolling parameter comprises determining at least one of: a pinchparameter, a natural scrolling parameter, a transfer function, aselection parameter, a dimensional parameter, a discrete parameter, acontinuous parameter, a scrolling speed parameter, or a scrollingdistance parameter. At step 912, a navigation command may be identifiedbased on the type of the input gesture, the control method, and thescrolling parameter. According to an aspect, applying the navigationcommand comprises moving at least a portion of the virtual object basedon a scrolling technique defined by the navigation command. The inputgesture corresponds to a midair wrist movement.

At step 914, the navigation command can be applied to the virtualobject. According to an aspect, the process 900 may further includedetermining a scrolling speed of the navigation command based on a speedor angle of wrist movement corresponding to the input gesture. Accordingto an aspect, the process 900 may further include generating, based onthe scrolling parameter, a momentum of scrolling through a scrollablelist of the virtual object in the shared artificial reality environmentaccording to the input gesture. According to an aspect, the process 900may further include selecting an item of the scrollable list based onthe navigation command and a double pinch input gesture.

FIG. 10 is a block diagram illustrating an exemplary computer system1000 with which aspects of the subject technology can be implemented. Incertain aspects, the computer system 1000 may be implemented usinghardware or a combination of software and hardware, either in adedicated server, integrated into another entity, or distributed acrossmultiple entities.

The computer system 1000 (e.g., server and/or client) includes a bus1008 or other communication mechanism for communicating information, anda processor 1002 coupled with the bus 1008 for processing information.By way of example, the computer system 1000 may be implemented with oneor more processors 1002. Each of the one or more processors 1002 may bea general-purpose microprocessor, a microcontroller, a Digital SignalProcessor (DSP), an Application Specific Integrated Circuit (ASIC), aField Programmable Gate Array (FPGA), a Programmable Logic Device (PLD),a controller, a state machine, gated logic, discrete hardwarecomponents, or any other suitable entity that can perform calculationsor other manipulations of information.

The computer system 1000 can include, in addition to hardware, code thatcreates an execution environment for the computer program in question,e.g., code that constitutes processor firmware, a protocol stack, adatabase management system, an operating system, or a combination of oneor more of them stored in an included memory 1004, such as a RandomAccess Memory (RAM), a flash memory, a Read-Only Memory (ROM), aProgrammable Read-Only Memory (PROM), an Erasable PROM (EPROM),registers, a hard disk, a removable disk, a CD-ROM, a DVD, or any othersuitable storage device, coupled to bus 1008 for storing information andinstructions to be executed by processor 1002. The processor 1002 andthe memory 1004 can be supplemented by, or incorporated in, specialpurpose logic circuitry.

The instructions may be stored in the memory 1004 and implemented in oneor more computer program products, i.e., one or more modules of computerprogram instructions encoded on a computer-readable medium for executionby, or to control the operation of, the computer system 1000, andaccording to any method well-known to those of skill in the art,including, but not limited to, computer languages such as data-orientedlanguages (e.g., SQL, dBase), system languages (e.g., C, Objective-C,C++, Assembly), architectural languages (e.g., Java, .NET), andapplication languages (e.g., PHP, Ruby, Perl, Python). Instructions mayalso be implemented in computer languages such as array languages,aspect-oriented languages, assembly languages, authoring languages,command line interface languages, compiled languages, concurrentlanguages, curly-bracket languages, dataflow languages, data-structuredlanguages, declarative languages, esoteric languages, extensionlanguages, fourth-generation languages, functional languages,interactive mode languages, interpreted languages, iterative languages,list-based languages, little languages, logic-based languages, machinelanguages, macro languages, metaprogramming languages, multiparadigmlanguages, numerical analysis, non-English-based languages,object-oriented class-based languages, object-oriented prototype-basedlanguages, off-side rule languages, procedural languages, reflectivelanguages, rule-based languages, scripting languages, stack-basedlanguages, synchronous languages, syntax handling languages, visuallanguages, wirth languages, and xml-based languages. Memory 1004 mayalso be used for storing temporary variable or other intermediateinformation during execution of instructions to be executed by theprocessor 1002.

A computer program as discussed herein does not necessarily correspondto a file in a file system. A program can be stored in a portion of afile that holds other programs or data (e.g., one or more scripts storedin a markup language document), in a single file dedicated to theprogram in question, or in multiple coordinated files (e.g., files thatstore one or more modules, subprograms, or portions of code). A computerprogram can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiplesites and interconnected by a communication network. The processes andlogic flows described in this specification can be performed by one ormore programmable processors executing one or more computer programs toperform functions by operating on input data and generating output.

The computer system 1000 further includes a data storage device 1006such as a magnetic disk or optical disk, coupled to bus 1008 for storinginformation and instructions. The computer system 1000 may be coupledvia input/output module 1010 to various devices. The input/output module1010 can be any input/output module. Exemplary input/output modules 1010include data ports such as USB ports. The input/output module 1010 isconfigured to connect to a communications module 1012. Exemplarycommunications modules 1012 include networking interface cards, such asEthernet cards and modems. In certain aspects, the input/output module1010 is configured to connect to a plurality of devices, such as aninput device 1014 and/or an output device 1016. Exemplary input devices1014 include a keyboard and a pointing device, e.g., a mouse or atrackball, by which a user can provide input to the computer system1000. Other kinds of input devices can be used to provide forinteraction with a user as well, such as a tactile input device, visualinput device, audio input device, or brain-computer interface device.For example, feedback provided to the user can be any form of sensoryfeedback, e.g., visual feedback, auditory feedback, or tactile feedback,and input from the user can be received in any form, including acoustic,speech, tactile, or brain wave input. Exemplary output devices 1016include display devices such as an LCD (liquid crystal display) monitor,for displaying information to the user.

According to one aspect of the present disclosure, the above-describedsystems can be implemented using a computer system 1000 in response tothe processor 1002 executing one or more sequences of one or moreinstructions contained in the memory 1004. Such instructions may be readinto memory 1004 from another machine-readable medium, such as datastorage device 1006. Execution of the sequences of instructionscontained in the main memory 1004 causes the processor 1002 to performthe process steps described herein. One or more processors in amulti-processing arrangement may also be employed to execute thesequences of instructions contained in the memory 1004. In alternativeaspects, hard-wired circuitry may be used in place of or in combinationwith software instructions to implement various aspects of the presentdisclosure. Thus, aspects of the present disclosure are not limited toany specific combination of hardware circuitry and software.

Various aspects of the subject matter described in this specificationcan be implemented in a computing system that includes a back endcomponent, e.g., such as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such back end, middleware, or front endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. The communication network can include, for example, any one ormore of a LAN, a WAN, the Internet, and the like. Further, thecommunication network can include, but is not limited to, for example,any one or more of the following network topologies, including a busnetwork, a star network, a ring network, a mesh network, a star-busnetwork, tree or hierarchical network, or the like. The communicationsmodules can be, for example, modems or Ethernet cards.

The computer system 1000 can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. Thecomputer system 1000 can be, for example, and without limitation, adesktop computer, laptop computer, or tablet computer. The computersystem 1000 can also be embedded in another device, for example, andwithout limitation, a mobile telephone, a PDA, a mobile audio player, aGlobal Positioning System (GPS) receiver, a video game console, and/or atelevision set top box.

The term “machine-readable storage medium” or “computer-readable medium”as used herein refers to any medium or media that participates inproviding instructions to the processor 1002 for execution. Such amedium may take many forms, including, but not limited to, non-volatilemedia, volatile media, and transmission media. Non-volatile mediainclude, for example, optical or magnetic disks, such as the datastorage device 1006. Volatile media include dynamic memory, such as thememory 1004. Transmission media include coaxial cables, copper wire, andfiber optics, including the wires that comprise the bus 1008. Commonforms of machine-readable media include, for example, floppy disk, aflexible disk, hard disk, magnetic tape, any other magnetic medium, aCD-ROM, DVD, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, a RAM, a PROM, an EPROM, aFLASH EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer can read. The machine-readable storage medium canbe a machine-readable storage device, a machine-readable storagesubstrate, a memory device, a composition of matter effecting amachine-readable propagated signal, or a combination of one or more ofthem.

As the user computing system 1000 reads XR data and provides anartificial reality, information may be read from the XR data and storedin a memory device, such as the memory 1004. Additionally, data from thememory 1004 servers accessed via a network, the bus 1008, or the datastorage 1006 may be read and loaded into the memory 1004. Although datais described as being found in the memory 1004, it will be understoodthat data does not have to be stored in the memory 1004 and may bestored in other memory accessible to the processor 1002 or distributedamong several media, such as the data storage 1006.

The techniques described herein may be implemented as method(s) that areperformed by physical computing device(s); as one or more non-transitorycomputer-readable storage media storing instructions which, whenexecuted by computing device(s), cause performance of the method(s); or,as physical computing device(s) that are specially configured with acombination of hardware and software that causes performance of themethod(s).

As used herein, the phrase “at least one of” preceding a series ofitems, with the terms “and” or “or” to separate any of the items,modifies the list as a whole, rather than each member of the list (i.e.,each item). The phrase “at least one of” does not require selection ofat least one item; rather, the phrase allows a meaning that includes atleast one of any one of the items, and/or at least one of anycombination of the items, and/or at least one of each of the items. Byway of example, the phrases “at least one of A, B, and C” or “at leastone of A, B, or C” each refer to only A, only B, or only C; anycombination of A, B, and C; and/or at least one of each of A, B, and C.

To the extent that the terms “include,” “have,” or the like is used inthe description or the claims, such term is intended to be inclusive ina manner similar to the term “comprise” as “comprise” is interpretedwhen employed as a transitional word in a claim. The word “exemplary” isused herein to mean “serving as an example, instance, or illustration.”Any embodiment described herein as “exemplary” is not necessarily to beconstrued as preferred or advantageous over other embodiments.

A reference to an element in the singular is not intended to mean “oneand only one” unless specifically stated, but rather “one or more.” Allstructural and functional equivalents to the elements of the variousconfigurations described throughout this disclosure that are known orlater come to be known to those of ordinary skill in the art areexpressly incorporated herein by reference and intended to beencompassed by the subject technology. Moreover, nothing disclosedherein is intended to be dedicated to the public regardless of whethersuch disclosure is explicitly recited in the above description.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of what may be claimed, but ratheras descriptions of particular implementations of the subject matter.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

The subject matter of this specification has been described in terms ofparticular aspects, but other aspects can be implemented and are withinthe scope of the following claims. For example, while operations aredepicted in the drawings in a particular order, this should not beunderstood as requiring that such operations be performed in theparticular order shown or in sequential order, or that all illustratedoperations be performed to achieve desirable results. The actionsrecited in the claims can be performed in a different order and stillachieve desirable results. As one example, the processes depicted in theaccompanying figures do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in the aspectsdescribed above should not be understood as requiring such separation inall aspects, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products. Othervariations are within the scope of the following claims.

What is claimed is:
 1. (canceled)
 2. A computer-implemented method forinteracting with virtual objects in an artificial reality environment,the method comprising: receiving sensed data comprising eye trackingdata specifying one or more relative coordinates, in the artificialreality environment, for an input gesture; identifying a position of avirtual object target of the input gesture based on the one or morerelative coordinates; receiving hand tracking data, defining positionsof one or more user hands, specifying the input gesture including apinch start point; determining, at a first time and based on a firstdistance between A) the pinch start point and B) a first current pinchpoint being below a threshold, that the input gesture is not a dragginggesture; determining, at a second time and based on a second distancebetween C) the pinch start point and D) a second current pinch pointbeing above the threshold, a type of the input gesture as being adragging gesture; simulating inertia for the dragging gesture bycalculating a transfer function indicating an amount of continuedmovement for the dragging gesture following a pinch release point in thehand tracking data; and applying, after a pinch release point thatoccurred in the hand tracking data and based on the simulated inertia,the dragging gesture to the virtual object target.
 3. Thecomputer-implemented method of claim 2, wherein the determining thefirst input gesture type comprises mapping E) a determined movement typein the hand tracking data comprising one of: flexion and extension,pronation and supination, or radial and ulnar to F) a correspondinggesture type.
 4. The computer-implemented method of claim 2, wherein thehand tracking data is received from a wrist mounted sensor.
 5. Thecomputer-implemented method of claim 2, wherein the hand tracking datais received from a visual sensor of a head-mounted device.
 6. Thecomputer-implemented method of claim 2, wherein the virtual objecttarget comprises at least one of: a scrollable list, a scrollableobject, a virtual area, or a highlighted virtual object.
 7. Thecomputer-implemented method of claim 2, wherein the determining that theinput gesture is not a dragging gesture and then determining a type ofthe input gesture as being a dragging gesture comprises identifying afirst motion by a first hand and a second motion by a second hand,wherein the second motion comprises a modification of the input gesturetriggered by the first motion.
 8. The computer-implemented method ofclaim 2, wherein the virtual object target is a virtual list; andwherein the applying the dragging gesture to the virtual list based onthe simulated inertia includes specifying the continued movement as,absent further interaction, a decreasing speed of scrolling of thevirtual list to a defined future stopping time according to the transferfunction.
 9. The computer-implemented method of claim 2, wherein thefirst input gesture type is a pinch input gesture.
 10. Thecomputer-implemented method of claim 2, wherein the identifying theposition of the virtual object target of the input gesture based on theone or more relative coordinates includes interpreting the input gesturerelative to a previously defined fixed neutral point.
 11. Thecomputer-implemented method of claim 2, wherein the transfer function isselected from a set of possible transfer functions based on adetermination of one or both of previous user scrolling patterns orcharacteristics of a scrollable list.
 12. A computer-readable storagemedium storing instructions that, when executed by a computing system,cause the computing system to perform a process for interacting withvirtual objects in an artificial reality environment, the processcomprising: identifying a position of a virtual object target of aninput gesture; receiving hand tracking data, defining positions of oneor more user hands, specifying the input gesture; determining, at afirst time and based on a first distance between points at which theinput gesture has been performed being below a threshold, that the inputgesture is not a dragging gesture; determining, at a second time andbased on a second distance between points at which the input gesture hasbeen performed being above the threshold, a type of the input gesture asbeing a dragging gesture; simulating inertia for the dragging gesture bycalculating a transfer function indicating an amount of continuedmovement for the dragging gesture following a pinch release point in thehand tracking data; and applying, after a pinch release point thatoccurred in the hand tracking data and based on the simulated inertia,the dragging gesture to the virtual object target.
 13. Thecomputer-readable storage medium of claim 12, wherein the processfurther comprises receiving sensed data comprising eye tracking dataspecifying one or more coordinates, in the artificial realityenvironment, for an input gesture; and wherein the identifying theposition of the virtual object target of the input gesture is based onthe one or more coordinates.
 14. The computer-readable storage medium ofclaim 12, wherein the determining the first input gesture type comprisesmapping A) a determined movement type in the hand tracking datacomprising one of: flexion and extension, pronation and supination, orradial and ulnar to B) a corresponding gesture type.
 15. Thecomputer-readable storage medium of claim 12, wherein the hand trackingdata is received from a visual sensor of a head-mounted device.
 16. Thecomputer-readable storage medium of claim 12, wherein the virtual objecttarget comprises at least one of: a scrollable list, a scrollableobject, a virtual area, or a highlighted virtual object.
 17. Thecomputer-implemented method of claim 2, wherein the virtual objecttarget is a virtual list; and wherein the applying the dragging gestureto the virtual list based on the simulated inertia includes specifyingthe continued movement as, absent further interaction, a decreasingspeed of scrolling of the virtual list to a defined future stopping timeaccording to the transfer function.
 18. A computing system forinteracting with virtual objects in an artificial reality environment,the computing system comprising: one or more processors; and one or morememories storing instructions that, when executed by the one or moreprocessors, cause the computing system to perform a process comprising:receiving sensed data comprising eye tracking data specifying one ormore coordinates, in the artificial reality environment, for an inputgesture; identifying a position of a virtual object target of the inputgesture based on the one or more coordinates; receiving hand trackingdata, defining positions of one or more user hands, specifying the inputgesture; determining, at a first time and based on a first distancebetween points at which the input gesture has been performed being belowa threshold, that the input gesture is not a dragging gesture;determining, at a second time and based on a second distance betweenpoints at which the input gesture has been performed being above thethreshold, a type of the input gesture as being a dragging gesture;applying, after a pinch release point that occurred in the hand trackingdata, the dragging gesture to the virtual object target.
 19. Thecomputing system of claim 18, wherein the identifying the position ofthe virtual object target of the input gesture based on the one or morecoordinates includes interpreting the input gesture relative to apreviously defined fixed neutral point.
 20. The computing system ofclaim 18, wherein the process further comprises simulating inertia forthe dragging gesture by calculating a transfer function indicating anamount of continued movement for the dragging gesture following a pinchrelease point in the hand tracking data; and wherein the applying thedragging gesture to the virtual object target is based on the simulatedinertia.
 21. The computing system of claim 20, wherein the transferfunction is selected from a set of possible transfer functions based ona determination of one or both of previous user scrolling patterns orcharacteristics of a scrollable list.